Hypothesis Testing

StartR Workshop

Maik Bieleke, PhD

University of Konstanz

November 24, 2024

Correlation

The correlation package

The correlation package is an easystats package focused on correlation analysis.

# install.packages("correlation")
library(correlation)

Computing single correlations

A single correlation can be determined with the cor_test() function.

# Correlation between players' value and overall attribute
cor_test(fifa, "Overall", "Value")

Parameter1 | Parameter2 |    r |       95% CI | t(17658) |         p
--------------------------------------------------------------------
Overall    |      Value | 0.56 | [0.55, 0.57] |    90.81 | < .001***

Observations: 17660

Plotting single correlations

We can plot the correlation using the plot() function.

r <- cor_test(fifa, "Overall", "Value")
plot(r)

Computing multiple correlations

We can compute correlations between multiple variables with the correlation() function.

# Select variables of interest
variables <- select(fifa, Overall, Potential, Value, Wage)

# Compute correlations
correlation(variables)

# Correlation Matrix (pearson-method)

Parameter1 | Parameter2 |    r |       95% CI | t(17658) |         p
--------------------------------------------------------------------
Overall    |  Potential | 0.71 | [0.70, 0.71] |   132.69 | < .001***
Overall    |      Value | 0.56 | [0.55, 0.57] |    90.81 | < .001***
Overall    |       Wage | 0.60 | [0.59, 0.61] |    99.55 | < .001***
Potential  |      Value | 0.51 | [0.50, 0.52] |    79.02 | < .001***
Potential  |       Wage | 0.48 | [0.47, 0.49] |    72.58 | < .001***
Value      |       Wage | 0.81 | [0.81, 0.82] |   183.96 | < .001***

p-value adjustment method: Holm (1979)
Observations: 17660

Plotting multiple correlations

We can use the summary()function to get a correlation matrix.

# Compute and save correlations
r <- correlation(variables)

# Generate correlation matrix
summary(r)

# Correlation Matrix (pearson-method)

Parameter |    Wage |   Value | Potential
-----------------------------------------
Overall   | 0.60*** | 0.56*** |   0.71***
Potential | 0.48*** | 0.51*** |          
Value     | 0.81*** |         |          

p-value adjustment method: Holm (1979)

Correlation plot

We can use the plot()function to plot the correlation matrix.

# Compute and save correlations and correlation matrix
r <- correlation(variables)
rmat <- summary(r)

# Generate correlation plot
plot(rmat)

Regression

Overview

There are numerous regression models available in R. We will focus on the Base R implementation of the linear regression model.

There are three aspects of regression:

model estimation: lm() function in base R
parameter evaluation: parameters package
model performance: performance package

The parameters and the performance package are part of the easystats package collection.

Simple regression

We can use the lm() function to estimate a simple linear regression model.

# estimate regression model with a single predictor: dv ~ iv
model <- lm(Wage ~ Overall, data = fifa)

# get the standard output
summary(model)


Call:
lm(formula = Wage ~ Overall, data = fifa)

Residuals:
   Min     1Q Median     3Q    Max 
-39708  -7903  -2542   5097 399598 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -88622.18     980.27  -90.41   <2e-16 ***
Overall       1527.74      15.35   99.55   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 16390 on 17658 degrees of freedom
Multiple R-squared:  0.3595,    Adjusted R-squared:  0.3595 
F-statistic:  9911 on 1 and 17658 DF,  p-value: < 2.2e-16

Model parameters and performance

We can use the parameters() and performance() to evaluate the model.

# parameters
parameters::parameters(model)

Parameter   | Coefficient |     SE |                 95% CI | t(17658) |      p
-------------------------------------------------------------------------------
(Intercept) |   -88622.18 | 980.27 | [-90543.60, -86700.76] |   -90.41 | < .001
Overall     |     1527.74 |  15.35 | [  1497.66,   1557.82] |    99.55 | < .001

# performance
performance::performance(model)

# Indices of model performance

AIC       |      AICc |       BIC |    R2 | R2 (adj.) |      RMSE |     Sigma
-----------------------------------------------------------------------------
3.929e+05 | 3.929e+05 | 3.929e+05 | 0.359 |     0.359 | 16387.478 | 16388.406

Model assumptions

We can use the check_model() function to check whether the assumptions of a linear regression are met.

performance::check_model(model)

Multiple regression

We can also use the lm() function to estimate a multiple regression model.

# estimate regression model with a single predictor: dv ~ iv
model2 <- lm(Wage ~ Overall + Value, data = fifa)
summary(model2)


Call:
lm(formula = Wage ~ Overall + Value, data = fifa)

Residuals:
    Min      1Q  Median      3Q     Max 
-132632   -3555    -908    2282  316273 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -3.045e+04  8.083e+02  -37.67   <2e-16 ***
Overall      5.315e+02  1.300e+01   40.89   <2e-16 ***
Value        1.810e-03  1.332e-05  135.83   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 11460 on 17657 degrees of freedom
Multiple R-squared:  0.6868,    Adjusted R-squared:  0.6867 
F-statistic: 1.936e+04 on 2 and 17657 DF,  p-value: < 2.2e-16

Comparing models

We can use the compare_performance() function to compare the performance of two or more models.

performance::compare_performance(model, model2, metrics = "common")

# Comparison of Model Performance Indices

Name   | Model |   AIC (weights) |   BIC (weights) |    R2 | R2 (adj.) |      RMSE
----------------------------------------------------------------------------------
model  |    lm | 3.9e+05 (<.001) | 3.9e+05 (<.001) | 0.359 |     0.359 | 16387.478
model2 |    lm | 3.8e+05 (>.999) | 3.8e+05 (>.999) | 0.687 |     0.687 | 11459.711

t-Tests

The `t-test()` function

The t-test() function has two basic forms:

Test	Wide Format	Long Format
One-sample	`t.test(x)`
Independent	`t.test(x, y)`	`t.test(y ~ x, data)`
Dependent	`t.test(x, y, paired = T)`	`t.test(y ~ x, data, paired = T)`

The function has various additional options, e.g.

var.equal for equal variances (TRUE or FALSE)
alternative for one-sided tests (less or more)
mu for the null hypothesis value (default is 0)
conf.level for the confidence interval (default is 0.95)

Single sample t-test

t-tests can be performed with the t.test() function in base R. If one variable is provided, a single sample t-test is performed.

# Test whether pre-training performance is different from 0
t.test(hiit$RSA_Pre)


    One Sample t-test

data:  hiit$RSA_Pre
t = 44.879, df = 21, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 4.786515 5.251667
sample estimates:
mean of x 
 5.019091

Independent t-test

# Compare pre-training performance between males and females
t.test(RSA_Pre ~ Sex, data = hiit)


    Welch Two Sample t-test

data:  RSA_Pre by Sex
t = -8.8367, df = 17.154, p-value = 8.54e-08
alternative hypothesis: true difference in means between group female and group male is not equal to 0
95 percent confidence interval:
 -1.1327500 -0.6963409
sample estimates:
mean in group female   mean in group male 
            4.561818             5.476364

Dependent t-test

# Compare pre- and post-training performance within individuals
t.test(hiit$RSA_Pre, hiit$RSA_Post1, paired = TRUE)


    Paired t-test

data:  hiit$RSA_Pre and hiit$RSA_Post1
t = 6.9213, df = 21, p-value = 7.733e-07
alternative hypothesis: true mean difference is not equal to 0
95 percent confidence interval:
 0.1255983 0.2334927
sample estimates:
mean difference 
      0.1795455

ANOVA

The afex and the emmeans package

afex (Analysis of Factorial EXperiments) is dedicated to analysis of variance.

# install.packages("afex")
library(afex)

emmeans (Estimated Marginal MEANS) is dedicated to post-hoc tests.

# install.packages("emmeans")
library(emmeans)

The `aov_4()` function

The aov_4() function is part of the afex package and can be used to estimate a variety of ANOVA models.

Generic format: aov_4(dv ~ iv_b + (iv_w | id), data)

dv is the dependent variable
iv_b are the between-subject predictor variables
iv_w are the within-subject predictor variables
id is the subject identifier variable
data is the data frame

The `emmeans()` function

The emmeans() function is part of the emmeans package. It has the following generic format:

Generic format: emmeans(model, ~ iv)

model is the model object
iv are the predictor variables

One-factorial ANOVA

Between-subject ANOVA

# Between-subject predictor: Sex
data <- dplyr::filter(hiit_long, Measure == "RSA" & Time == "Pre")
afex::aov_4(Value ~ Sex + (1 | ID), data)

Anova Table (Type 3 tests)

Response: Value
  Effect    df  MSE         F  ges p.value
1    Sex 1, 20 0.06 78.09 *** .796   <.001
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1

Within-subject ANOVA

# Within-subject predictor: Time
data <- dplyr::filter(hiit_long, Measure == "RSA")
afex::aov_4(Value ~ 1 + (Time | ID), data)

Anova Table (Type 3 tests)

Response: Value
  Effect          df  MSE       F  ges p.value
1   Time 1.30, 27.20 0.04 7.16 ** .020    .008
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1

Sphericity correction method: GG

Pairwise comparisons

# Within-subject predictor: Time
data <- dplyr::filter(hiit_long, Measure == "RSA")
model <- afex::aov_4(Value ~ 1 + (Time | ID), data)

# Esimate the marginal means
emm <- emmeans::emmeans(model, ~ Time)
emm

 Time  emmean    SE df lower.CL upper.CL
 Pre     5.02 0.112 21     4.79     5.25
 Post1   4.84 0.118 21     4.59     5.09
 Post2   4.97 0.119 21     4.73     5.22

Confidence level used: 0.95

# Conduct pairwise comparisons
pairs(emm)

 contrast      estimate     SE df t.ratio p.value
 Pre - Post1     0.1795 0.0259 21   6.921  <.0001
 Pre - Post2     0.0459 0.0601 21   0.764  0.7289
 Post1 - Post2  -0.1336 0.0548 21  -2.440  0.0589

P value adjustment: tukey method for comparing a family of 3 estimates

Mixed ANOVA

# Between-subject predictor: Sex, Within-subject predictor: Time
data <- dplyr::filter(hiit_long, Measure == "RSA")
afex::aov_4(Value ~ Sex + (Time | ID), data)

Anova Table (Type 3 tests)

Response: Value
    Effect          df  MSE         F  ges p.value
1      Sex       1, 20 0.16 90.01 *** .770   <.001
2     Time 1.26, 25.24 0.04   6.97 ** .081    .010
3 Sex:Time 1.26, 25.24 0.04      0.43 .005    .566
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1

Sphericity correction method: GG

Simple effects

# Between-subject predictor: Sex, Within-subject predictor: Time
data <- dplyr::filter(hiit_long, Measure == "RSA")
model <- afex::aov_4(Value ~ Sex + (Time | ID), data)

# Esimate the marginal means
emm <- emmeans::emmeans(model, ~ Sex * Time)

# Estimate simple effects
emmeans::joint_tests(emm, by = "Time")

Time = Pre:
 model term df1 df2 F.ratio p.value
 Sex          1  20  78.087  <.0001

Time = Post1:
 model term df1 df2 F.ratio p.value
 Sex          1  20  99.553  <.0001

Time = Post2:
 model term df1 df2 F.ratio p.value
 Sex          1  20  43.946  <.0001

Hypothesis Testing

Correlation

The correlation package

Computing single correlations

Plotting single correlations

Computing multiple correlations

Plotting multiple correlations

Correlation plot

Regression

Overview

Simple regression

Model parameters and performance

Model assumptions

Multiple regression

Comparing models

t-Tests

The t-test() function

Single sample t-test

Independent t-test

Dependent t-test

ANOVA

The afex and the emmeans package

The aov_4() function

The emmeans() function

One-factorial ANOVA

Pairwise comparisons

Mixed ANOVA

Simple effects

The `t-test()` function

The `aov_4()` function

The `emmeans()` function